Computer systems utilized for business system generate messages which audit user access, service errors, and other critical information about the operation of the systems. These messages are recorded in a log, managed by the computing system and therefore called log messages. Traditionally log messages are recorded to files on the local file system, or in the case of syslog enabled systems, can be redirected and stored on the local file system of separate system. Computing systems based on Microsoft Windows record log messages to the local file system via the Windows Event Log.
Recent industry and government regulations such as the Payment Card Industry Data Security Standard (PCI DSS), Sarbanes-Oxley Act (SOX), Health Insurance Portability and Accountability Act (HIPAA), and the Gramm-Leach-Bliley Act (GLBA), require that log data be collected, regularly reviewed, and securely archived. To meet the requirements of these regulations log message files must be archived for up to seven (7) years. For large organizations or organizations with specialized operations, the volume of log messages generated may require storage volumes approaching petabytes (PB) of data. This has generally resulted in significant capital investment, staffing expense and operational complexity to provide secure and reliable storage for this length of time.
Of particular interest is the complexity and cost involved in maintaining large volumes of digital information, such as log data, over the number of years required by these regulations. Simply storing log data on computer media (e.g. hard disks or tape) is prone to media deterioration and failure resulting in the loss of the data. Computer storage arrays, regardless of media, reliably handle the failure of a fraction of the total number of media devices, but over time on the order of five (5) or more year, all of the media devices in the storage array have passed the manufacturer specified duty cycle and are likely to fail in a way that is not recoverable by the storage array. The effort of maintaining large volumes of digital data reliably over this time frame therefore requires continuous investment in terms of hardware and expertise.
Current solutions to managing log data are available as part of the base operating system of computing systems, or as products offered for deployment on the customer's private network. Overall, these solutions require significant staff expertise or capital investment to deploy and maintain in a way that meets regulatory requirements, which usually must be borne by the customer alone.